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Summary: 

In  this  project,  we  addressed  some  of  the  most  challenging,  yet  seldom  studied,  open  problems 
in  image  and  video  enhancement  of  interest  to  the  Air  Force;  namely,  the  removal  of 
disturbances  due  to  demanding  physical  and  environmental  conditions.  We  considered 
degradations  of  interest  that  can  be  caused  by  a  number  of  different  physical  phenomena  (often 
occurring  simultaneously)  which  are  commonly  encountered  in  practice.  Such  degradations 
include 

•  air  turbulence,  resulting  in  pictures  which  suffer  from  random,  spatially  varying,  blur, 

•  complex  motion  blur  due  to  the  non-stationarity  of  the  imaging  platform  or  the  objects 
being  imaged,  resulting  in  pictures  with  highly  variable  blur  characteristics,  and 

•  non-ideal  lighting  and  depth-of-focus  conditions,  resulting  in  images  containing  variable 
blur  and  significant  levels  of  noise. 

To  address  these  problems,  we  proposed  a  novel  adaptive  framework.  Specifically,  in  the  case 
when  multiple  images  are  available  (such  as  turbulence  imaging),  we  automatically  select 
isoplanatically  blurred  regions  (i.e.,  those  blurred  with  short-support  point-spread  functions,) 
which  can  approximately  be  viewed  as  space-invariant.  From  these  a  sharp  image  can  be 
estimated  through  adaptive  non-parametric  fusion,  followed  by  deconvolution.  Experiments 
proved  that  this  method  could  significantly  reduce  the  effect  of  turbulence  in  the  acquired 
imagery.  More  generally,  a  special  case  of  this  technique  is  also  applicable  when  only  a  single 
image  is  available,  and  hence  the  general  idea  is  applicable  to  the  other  scenarios  as  well. 
Finally,  although  spatial  deblurring  is  relatively  well-understood  by  assuming  that  the  blur 
kernel  is  shift-invariant,  spatially  varying  motion  blur  is  not  so  when  we  attempt  to  deconvolve 
this  motion  blur  on  a  frame-by-frame  basis:  this  is  because,  in  general,  videos  include  complex, 
multi-layer  transitions.  Indeed,  we  face  an  exceedingly  difficult  problem  in  motion  deblurring  of  a 
single  frame  when  the  scene  contains  multiple  motions  or  occlusions.  Instead  of  deblurring 
video  frames  individually,  a  fully  3-D  deblurring  method  is  proposed  here.  Most  importantly,  due 
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to  its  inherent  locally  adaptive  nature,  the  proposed  approach  is  capable  of  automatically 
deblurring  the  parts  of  the  sequence  which  are  motion  blurred,  without  segmentation,  and 
without  adversely  affecting  the  rest  of  the  spatiotemporal  domain  where  such  blur  is  not  present. 

Detailed  Accomplishments: 

•  Removing  Atmospheric  Turbulence  via  Space-Invariant 
Deconvolution: 

o  To  correct  geometric  distortion  and  reduce  space  and  time-varying  blur,  a  new 
approach  is  proposed  capable  of  restoring  a  single  high-quality  image  from  a 
given  image  sequence  distorted  by  atmospheric  turbulence.  This  approach 
reduces  the  space  and  time-varying  deblurring  problem  to  a  shift  invariant  one.  It 
first  registers  each  frame  to  suppress  geometric  deformation  through 
B-spline-based  nonrigid  registration.  Next,  a  temporal  regression  process  is 
carried  out  to  produce  an  image  from  the  registered  frames,  which  can  be  viewed 
as  being  convolved  with  a  space  invariant  near-diffraction-limited  blur.  Finally,  a 
blind  deconvolution  algorithm  is  implemented  to  deblur  the  fused  image, 
generating  a  final  output.  Experiments  using  real  data  illustrate  that  this  approach 
can  effectively  alleviate  blur  and  distortions,  recover  details  of  the  scene,  and 
significantly  improve  visual  quality 
o  Also  see  related  talk  .  Project  page  .  Software  package 


•  Estimating  Spatially  Varying  Defocus  Blur  from  A  Single  Image 

o  Estimating  the  amount  of  blur  in  a  given  image  is  important  for  computer  vision 
applications.  More  specifically,  the  spatially  varying  defocus 
point-spread-functions  (PSFs)  over  an  image  reveal  geometric  information  of  the 
scene,  and  their  estimate  can  also  be  used  to  recover  an  all-in-focus  image.  A 
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PSF  for  a  defocus  blur  can  be  specified  by  a  single  parameter  indicating  its  scale. 
Most  existing  algorithms  can  only  select  an  optimal  blur  from  a  finite  set  of 
candidate  PSFs  for  each  pixel.  Some  of  those  methods  require  a  coded  aperture 
filter  inserted  in  the  camera.  In  this  paper,  we  present  an  algorithm  estimating  a 
defocus  scale  map  from  a  single  image,  which  is  applicable  to  conventional 
cameras.  This  method  is  capable  of  measuring  the  probability  of  local  defocus 
scale  in  the  continuous  domain.  It  also  takes  smoothness  and  color  edge 
information  into  consideration  to  generate  a  coherent  blur  map  indicating  the 
amount  of  blur  at  each  pixel.  Simulated  and  real  data  experiments  illustrate 
excellent  performance  and  its  successful  applications  in  foreground/background 
segmentation. 


•  Robust  Flash  Denoising/Deblurring  by  Iterative  Guided  Filtering 

o  A  practical  problem  addressed  recently  in  computational  photography  is  that  of 
producing  a  good  picture  of  a  poorly  lit  scene.  The  consensus  approach  for 
solving  this  problem  involves  capturing  two  images  and  merging  them.  In 
particular,  using  a  flash  produces  one  (typically  high  signal-to-noise  ratio  [SNR]) 
image  and  turning  off  the  flash  produces  a  second  (typically  low  SNR)  image.  In 
this  article,  we  present  a  novel  approach  for  merging  two  such  images.  Our 
method  is  a  generalization  of  the  guided  filter  approach  of  He  et  al.,  significantly 
improving  its  performance.  In  particular,  we  analyze  the  spectral  behavior  of  the 
guided  filter  kernel  using  a  matrix  formulation,  and  introduce  a  novel  iterative 
application  of  the  guided  filter.  These  iterations  consist  of  two  parts:  a  nonlinear 
anisotropic  diffusion  of  the  noisier  image,  and  a  nonlinear  reaction-diffusion 
(residual)  iteration  of  the  less  noisy  one.  The  results  of  these  two  processes  are 
combined  in  an  unsupervised  manner.  We  demonstrate  that  the  proposed 
approach  outperforms  state-of-the-art  methods  for  both  flash/no-flash  denoising, 
and  deblurring. 

o  Also  see  related  project  page. 
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•  Nonlocal  Image  Editing 

o  We  introduce  a  new  image  editing  tool  based  on  the  spectrum  of  a  global  filter 
computed  from  image  affinities.  Recently,  it  has  been  shown  that  the  global  filter 
derived  from  a  fully  connected  graph  representing  the  image  can  be 
approximated  using  the  Nystrom  extension.  This  filter  is  computed  by 
approximating  the  leading  eigenvectors  of  the  filter.  These  orthonormal 
eigenfunctions  are  highly  expressive  of  the  coarse  and  fine  details  in  the 
underlying  image,  where  each  eigenvector  can  be  interpreted  as  one  scale  of  a 
data-dependent  multiscale  image  decomposition.  In  this  filtering  scheme,  each 
eigenvalue  can  boost  or  suppress  the  corresponding  signal  component  in  each 
scale.  Our  analysis  shows  that  the  mapping  of  the  eigenvalues  by  an  appropriate 
polynomial  function  endows  the  filter  with  a  number  of  important  capabilities, 
such  as  edge-aware  sharpening,  denoising,  tone  manipulation,  and  abstraction, 
to  name  a  few.  Furthermore,  the  edits  can  be  easily  propagated  across  the 
image. 

o  Also  see  related  (Project  Weboaae) 
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•  A  General  Framework  for  Regularized,  Similarity-based  Image 
Restoration 

o  Any  image  can  be  represented  as  a  function  defined  on  a  weighted  graph,  in 
which  the  underlying  structure  of  the  image  is  encoded  in  kernel  similarity  and 
associated  Laplacian  matrices.  We  developed  an  iterative  graph-based 
framework  for  image  restoration  based  on  a  new  definition  of  the  normalized 
graph  Laplacian.  We  proposed  a  cost  function,  which  consists  of  a  new  data 
fidelity  term  and  regularization  term  derived  from  the  specific  definition  of  the 
normalized  graph  Laplacian.  The  normalizing  coefficients  used  in  the  definition  of 
the  Laplacian  and  associated  regularization  term  are  obtained  using  fast 
symmetry  preserving  matrix  balancing.  This  results  in  some  desired  spectral 
properties  for  the  normalized  Laplacian  such  as  being  symmetric,  positive 
semidefinite,  and  returning  zero  vector  when  applied  to  a  constant  image.  The 
proposed  approach  is  general  in  the  sense  that  we  have  shown  its  effectiveness 
for  different  restoration  problems,  including  deblurring,  denoising,  and 
sharpening.  Experimental  results  verify  the  effectiveness  of  the  proposed 
algorithm  on  both  synthetic  and  real  examples. 
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•  Pushing  back  the  limits  of  Raman  imaging  by  coupling 
super-resolution  and  chemometrics  for  aerosols 
characterization 

o  The  increasing  interest  in  nanoscience  in  many  research  fields  like 
physics,  chemistry,  and  biology,  including  the  environmental  fate  of  the 
produced  nano-objects,  requires  instrumental  improvements  to  address 
the  sub-micrometric  analysis  challenges.  The  originality  of  our  approach 
is  to  use  both  the  super-resolution  concept  and  multivariate  curve 
resolution  (MCR-ALS)  algorithm  in  confocal  Raman  imaging  to  surmount 
its  instrumental  limits  and  to  characterize  chemical  components  of 
atmospheric  aerosols  at  the  level  of  the  individual  particles.  We 
demonstrate  the  possibility  to  go  beyond  the  diffraction  limit  with  this 
algorithmic  approach.  Indeed,  the  spatial  resolution  is  improved  by  65%  to 
achieve  200  nm  for  the  considered  far-field  spectrophotometer.  A 
multivariate  curve  resolution  method  is  then  coupled  with  super-resolution 
in  order  to  explore  the  heterogeneous  structure  of  submicron  particles  for 
describing  physical  and  chemical  processes  that  may  occur  in  the 
atmosphere.  The  proposed  methodology  provides  new  tools  for 
sub-micron  characterization  of  heterogeneous  samples  using  far-field  (i.e. 
conventional)  Raman  imaging  spectrometer. 


100  low-resolution  Images 
ofSi@  520,7cm-i 


Super-resolution  image 
of  Si  @  520,7  cm‘^ 
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Additional  Synergistic  Activities  and  Accomplishments 


•  Distinguished  Lecturer,  IEEE  Signal  Processing  Society 

•  Elected  Fellow  of  the  IEEE 

•  Supported  and  Graduated  5  Ph.D.  Students,  2  M.S.  Students 

•  Presented  many  plenary  and  keynote  speeches,  including: 

o  (2015)  Plenary,  International  Conference  on  Multimedia  and  Exposition,  Torino 
o  (2015)  Keynote,  SPIE  Conference  on  Digital  Image  Processing,  San  Francisco 
o  (2014)  Keynote,  Workshop  on  Co-design  of  hybrid  imaging  systems,  Paris 
o  (2014)  P/ena/y,  SPIE  Optics  and  Photonics  Symposium,  San  Diego 
o  (2014)  Plenary,  Technion’s  TCE  Symposium,  Haifa 
o  (2013)  Plenary,  Picture  Coding  Symposium 
o  (2012)  Plenary,  Mathematics  and  Image  Analysis  Conference,  Paris 
o  (2010)  Keynote,  Pacific  Rim  Symp.  on  Image  and  Video  Technology,  Singapore 
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